| 1 |
{# |
| 2 |
OLMo-2 reference template from the live HF tokenizer_config: |
| 3 |
|
| 4 |
{{ bos_token }}{% for message in messages %} |
| 5 |
{% if message['role'] == 'system' %}<|system|>\n...{% elif ... %} |
| 6 |
|
| 7 |
The important shape is: |
| 8 |
- prepend BOS (`<|endoftext|>`) |
| 9 |
- system/user turns are raw role markers plus trailing newline |
| 10 |
- assistant turns close with `<|endoftext|>` |
| 11 |
- the generation prompt is `<|assistant|>\n` |
| 12 |
#} |
| 13 |
{{- '<|endoftext|>' -}} |
| 14 |
{%- for message in messages -%} |
| 15 |
{%- if message['role'] == 'system' -%} |
| 16 |
<|system|> |
| 17 |
{{ message['content'] }} |
| 18 |
{%- elif message['role'] == 'user' -%} |
| 19 |
<|user|> |
| 20 |
{{ message['content'] }} |
| 21 |
{%- elif message['role'] == 'assistant' -%} |
| 22 |
<|assistant|> |
| 23 |
{{ message['content'] }}<|endoftext|> |
| 24 |
{%- endif -%} |
| 25 |
{% endfor -%} |
| 26 |
{%- if add_generation_prompt -%} |
| 27 |
<|assistant|> |
| 28 |
{%- endif -%} |
| 29 |
|