.Net [ChatRoles] 1. Convert SK templating engine chat role syntax to intermediate SK format #3250

As an engineer working on SK,
I want templating engines to generate an intermediate format for chat roles,
So that subsequent components (such as semantic functions or connectors) can recognize it and map it to chat messages or history

Scope:

1. Create ADR and present it to the team.
2. Collect feedback and implement the conversion functionality in the 'BasicPromptTemplateEngine' class.

Context

Today, various templating engines have their unique syntax and may represent chat completion roles in a manner specific to the engine's syntax:

• jinja2 template used by promptflow- https://github.com/microsoft/promptflow/blob/main/examples/flows/chat/basic-chat/chat.jinja2

• handlebars template used by guidance - https://github.com/guidance-ai/guidance/blob/23d0ba12720d09bb87b520d6c84462857f5dfcec/notebooks/api_examples/library/role.ipynb#L10

  

To prevent the handling of templating engine-specific syntax in other SK components besides the prompt template, we need a mechanism to map templating engine-specific syntax to the SK format. For example, if the SK prompt uses syntax like:

• Handlebars: "{{#system}}You are a banker.{{/system}}"
• Jinja2: "system: You are a banker"

Then it should be converted to the SK format, so that the resulting prompt, as it will be described in the ADR, looks like "<role:system>You are a banker.<role:system>." This way, the other SK components (connectors?) participating in the semantic function execution can rely on a well-defined format to extract those roles from the rendered prompt and map them to chat messages for chat completion APIs.