TraceQL recording rules: inconsistencies between labeling conventions between mimir and tempo cause err-mimir-label-invalid errors

### What happened? Because tempo has a dot notation convention for labels, and mimir has an underscore convention, when we try to create Traceql recording rule (https://github.com/grafana/grafana/issues/104289) that includes a label with a dot (or other invalid mimir label character), then the recording rule throws err-mimir-label-invalid. For example this TraceQL will succeed: ` { resource.service.name="my-most-excellent-service"} | max_over_time(span:duration) by (name) ` However these TraceQL will throw: ` { resource.service.name="my-most-excellent-service"} | max_over_time(span:duration) by (.span.kind) ` ` { resource.service.name="my-most-excellent-service"} | max_over_time(span:duration) by (span:kind) ` ### What did you expect to happen? I think the most elegant solution here would be for TraceQL to support some kind of label rename at query time, then I could explicitly provide a mimir-safe label name; I didn't find that or similar feature request on the Tempo backlog... maybe a less elegant solution would be to coerce invalid characters to underscores? Any thoughts/advice on the matter will be appreciated, thanks! Ryan ### Did this work before? N/A ### How do we reproduce it? 1. Create a TraceQL recording rule a la https://github.com/grafana/grafana/issues/104289 2. Use a query like so ` { resource.service.name="my-most-excellent-service"} | max_over_time(span:duration) by (span:kind) ` 3. Observe the error: ` remote write failed: series was rejected: received a series with an invalid label: 'resource.service.name' series: 'test_recording_rule_delete_me_DEVOPS_231715{jenkins_computer_name="{{ $labels.span.jenkins.computer.name }}", resource.service.name="\"trident-controller\""}' (err-mimir-label-invalid) ` ### Is the bug inside a dashboard panel? N/A ### Environment (with versions)? Grafana: 12.1.1 OS: Windows Browser: Chrome ### Grafana platform? Kubernetes ### Datasource(s)? _No response_