AI General Thread

[Legend]

For some reason, running local models feels goofier than it should be. And I'm not sure why. Like for some reason, prompt adherence is worse when I'm using different Android app.

Not sure if they're formatting the requests differently, or passing different default values for the model.

Different sampling method/temperature?

[the-pi-guy]

Different sampling method/temperature?

Most of the apps let you change the temperature, top K, top P. 

I'm leaning towards formatting being different, which would be harder to fix as an end user.

[the-pi-guy]

LM Studio has so many options you can mess with. 

You can even set how the context window is managed (truncate middle, rolling).  

[Legend]

LM Studio has so many options you can mess with.

You can even set how the context window is managed (truncate middle, rolling). 

Is there a way to run two models in parallel so that the next token can be sampled using both their outputs? Like averaged or multiplied or whatever?

[the-pi-guy]

Is there a way to run two models in parallel so that the next token can be sampled using both their outputs? Like averaged or multiplied or whatever?

Not that I know of.

[the-pi-guy]

Something that I've really wanted to do is combine two bots.

I don't think the bots are actually smart enough to do this.
I just tried something with copilot, and it kind of can do it as a single instance. But it felt sloppy.

But it would be cool to have two bots - for example have one of them be a Dungeon Master and have one be a player. Where the latter one has no knowledge of the other's thoughts.

You could probably do it with a single instance, but I think chances are good that the model would get confused. 

The way I'm thinking that I would try it, would still technically be a single instance, but just having a program to carefully cull different parts of the context to simulate each bot.