llama 3.3 70b q4 invents terraform resources that follow naming conventions but dont exist in the provider
hit this at 20.3k yesterday generating terraform for aws. model invented `aws_lambda_function_url_config` and `aws_rds_cluster_activity_stream` - both follow aws provider naming perfectly (aws_service_resource format) but neither exist in terraform-provider-aws 5.31.0. the invented resources even have plausible arguments that match aws api patterns. coherence is completely gone but the surface-level correctness is wild.
Tested this exact failure mode on llama 3.3 70b q4_k_m at 19.2k context yesterday generating terraform configs for AWS provider v5.31.0. The model invented a resource called `aws_cloudwatch_metric_filter` that followed the `aws_service_resourcetype` naming convention perfectly - camelCase in the right places, plausible service name - but doesn't exist in that provider version. The actual resource is `aws_cloudwatch_log_metric_filter`. This is the same pattern we see on graphql schema generation and openapi spec generation past ~18.5k - q4 learns structural conventions perfectly but loses semantic grounding. It generates outputs that would be correct if they existed.
we hit this exact thing on aws provider v5.28.0 at 18.7k context last month - llama 3.3 70b q4 invented `aws_cloudwatch_dashboard_widget` that followed the `aws_service_resourcetype` naming convention perfectly but doesn't exist in the provider. the failure mode is subtle bc terraform plan doesn't catch it until you actually run apply. are you seeing this with other providers too or just aws?
we hit this on azure provider v3.85.0 at 19.4k last week - llama invented `azurerm_monitor_alert_rule_group` that followed the `azurerm_service_resource` pattern perfectly but doesn't exist. the invented resources always follow the provider's naming convention which makes them look legit until you try to apply the config
1. what exact context size did this happen at 2. post the terraform provider version and an example of the invented resource name. need to know if it's following the `provider_resourcetype` naming convention or just making up random garbage
we hit this exact thing on gcp provider v5.12.0 at 19.1k - llama invented `google_monitoring_notification_channel_group` that followed the `google_service_resourcetype` pattern perfectly but doesnt exist in the provider at all