[{"data":1,"prerenderedAt":260},["ShallowReactive",2],{"werk-list-en":3},[4,104,172],{"id":5,"title":6,"body":7,"date":91,"description":92,"draft":93,"extension":94,"image":91,"imageAlt":91,"meta":95,"metric":96,"metricLabel":97,"navigation":98,"order":99,"path":100,"seo":101,"stem":102,"__hash__":103},"en\u002Fwerk\u002Frag.md","RAG in production",{"type":8,"value":9,"toc":83},"minimark",[10,17,22,25,29,51,55,62,65,68,72],[11,12,13],"blockquote",{},[14,15,16],"p",{},"This case is described anonymously: no client name, no code and no screenshots.",[18,19,21],"h2",{"id":20},"problem","Problem",[14,23,24],{},"A search feature that answers from your own documents is quick to build but hard to judge. Without a fixed measurement you do not know after each change whether the answers got better or worse. You end up testing by feel.",[18,26,28],{"id":27},"what-i-did","What I did",[30,31,32,36,39,42],"ul",{},[33,34,35],"li",{},"Split documents into passages and stored an embedding per passage, so a question finds the passages closest to it in meaning.",[33,37,38],{},"Built a reranker that scores the retrieved passages against the question again. It is still switched off in production: it needs its own model and the live environment cannot run that yet. The embedding search does run live.",[33,40,41],{},"Built an evaluation with a fixed set of questions for which the right passage is known. Every change now gets a score.",[33,43,44,45,50],{},"Tested a summary index as an alternative and threw it away again. That story is in ",[46,47,49],"a",{"href":48},"\u002Fen\u002Fwriting\u002Fsummary-index","a separate article",".",[18,52,54],{"id":53},"what-i-measured","What I measured",[14,56,57,58,50],{},"The evaluation uses MRR (mean reciprocal rank). If the right passage is in position 1, that counts as 1. Position 2 counts as 0.5, position 3 as 0.33. The average over all questions is the score. The score came out at ",[59,60,61],"strong",{},"0.950",[14,63,64],{},"To be honest about it: that score was the same with and without the reranker. The measurement looked at the position of the passage after the passages had been sorted again, so it never saw the reranker's order. The number therefore says something about finding passages, not about the reranker.",[14,66,67],{},"The summary index cost 7.7 seconds per file, against 0.25 seconds for embedding. That is roughly 30 times slower.",[18,69,71],{"id":70},"what-i-learned","What I learned",[30,73,74,77,80],{},[33,75,76],{},"Measure first, build second. A fixed score ends the discussion about what \"feels better\".",[33,78,79],{},"A measurement can be blind to exactly the part you want to judge. Check what the number does and does not see.",[33,81,82],{},"Throwing something away after measuring it is a result too.",{"title":84,"searchDepth":85,"depth":85,"links":86},"",2,[87,88,89,90],{"id":20,"depth":85,"text":21},{"id":27,"depth":85,"text":28},{"id":53,"depth":85,"text":54},{"id":70,"depth":85,"text":71},null,"Document search with embeddings, and an evaluation with fixed scores instead of testing by feel.",false,"md",{},"MRR 0.950","on a fixed set of evaluation questions",true,1,"\u002Fwerk\u002Frag",{"title":6,"description":92},"werk\u002Frag","crmO0g_oB38E7GVS-RMwz0eDcT20XTLE4UjirJsTxxc",{"id":105,"title":106,"body":107,"date":91,"description":166,"draft":93,"extension":94,"image":91,"imageAlt":91,"meta":167,"metric":91,"metricLabel":91,"navigation":98,"order":85,"path":168,"seo":169,"stem":170,"__hash__":171},"en\u002Fwerk\u002Fai-agents.md","Working with AI agents in a team",{"type":8,"value":108,"toc":161},[109,111,114,116,148,150],[18,110,21],{"id":20},[14,112,113],{},"An AI agent that writes code is fast, but it is just as fast at doing something you do not want: committing without being asked, deleting files or following its own style. In a team there is a second problem. Everyone makes their own agreements with the agent, so the result differs per developer.",[18,115,28],{"id":27},[30,117,118,124,130,136,142],{},[33,119,120,123],{},[59,121,122],{},"Shared rules."," The team's agreements live in one repository that every developer loads. Personal preferences are kept apart and win when they conflict.",[33,125,126,129],{},[59,127,128],{},"A hook as a safety net."," An agent can ignore a rule written in text. So a hook blocks dangerous git commands before they run.",[33,131,132,135],{},[59,133,134],{},"A model per task."," Mechanical work goes to a small model, normal research to a mid-size model, and only choices where judgment decides the outcome go to the largest model.",[33,137,138,141],{},[59,139,140],{},"Several issues in parallel."," One session divides the work and starts a separate agent per issue.",[33,143,144,147],{},[59,145,146],{},"A fixed verification step."," An agent's work is only done once it has been checked: lint, the full test suite and a check of the diff against the rules.",[18,149,71],{"id":70},[30,151,152,155,158],{},[33,153,154],{},"A rule that only exists as text is a request. What must never happen belongs in a hook.",[33,156,157],{},"Rules grow out of mistakes. Every correction gets a fixed place, so the same mistake does not come back.",[33,159,160],{},"Working in parallel only saves time if the check at the end is as strict as it is for a single issue.",{"title":84,"searchDepth":85,"depth":85,"links":162},[163,164,165],{"id":20,"depth":85,"text":21},{"id":27,"depth":85,"text":28},{"id":70,"depth":85,"text":71},"Shared rules, a hook that blocks dangerous git commands, a model per task and a fixed verification step.",{},"\u002Fwerk\u002Fai-agents",{"title":106,"description":166},"werk\u002Fai-agents","QZ7Vc3I47yvg8w3bofAF1OPFv80x2YylJgmcheah2Cg",{"id":173,"title":174,"body":175,"date":91,"description":253,"draft":93,"extension":94,"image":91,"imageAlt":91,"meta":254,"metric":91,"metricLabel":91,"navigation":98,"order":255,"path":256,"seo":257,"stem":258,"__hash__":259},"en\u002Fwerk\u002Fself-hosting.md","Self-hosting on a NAS",{"type":8,"value":176,"toc":246},[177,179,187,189,207,210,214,221,224,226,233,235],[18,178,21],{"id":20},[14,180,181,182,186],{},"I wanted to host my own apps myself: to know where the data lives and to understand what happens between a ",[183,184,185],"code",{},"git push"," and a working site. If you host it yourself, you are also responsible for what is exposed to the internet.",[18,188,28],{"id":27},[30,190,191,194,201,204],{},[33,192,193],{},"Apps run as Docker containers on a NAS.",[33,195,196,197,200],{},"The code lives on my own Forgejo. Deploying is a ",[183,198,199],{},"git pull"," on the NAS and rebuilding the container.",[33,202,203],{},"A reverse proxy (Nginx Proxy Manager) handles the domains and the Let's Encrypt certificates.",[33,205,206],{},"I also have some experience with Kubernetes and ArgoCD.",[14,208,209],{},"This site runs the same way.",[18,211,213],{"id":212},"the-security-mistake","The security mistake",[14,215,216,217,220],{},"One of my own sites was served by an nginx container with the default configuration, with the whole repository as the web root. That made the ",[183,218,219],{},".git"," folder, the README and the compose file downloadable by anyone. I found it myself while looking into a report about a redirect error.",[14,222,223],{},"The fix: my own nginx configuration outside the web root, which returns a 404 for dotfiles and other non-site files. The redirect error came from the same default configuration and was fixed in the same change.",[18,225,54],{"id":53},[14,227,228,229,232],{},"After the change I checked from the outside: a request for ",[183,230,231],{},"\u002F.git\u002Fconfig"," returns a 404 and a redirect stays on https. This site goes one step further: the container only holds the build output, so the repository is not even in the place that gets served.",[18,234,71],{"id":70},[30,236,237,240,243],{},[33,238,239],{},"A default configuration is made to work, not to be safe.",[33,241,242],{},"Serve the build, not the repository.",[33,244,245],{},"Check from the outside what is reachable. From the inside everything looks fine.",{"title":84,"searchDepth":85,"depth":85,"links":247},[248,249,250,251,252],{"id":20,"depth":85,"text":21},{"id":27,"depth":85,"text":28},{"id":212,"depth":85,"text":213},{"id":53,"depth":85,"text":54},{"id":70,"depth":85,"text":71},"My own apps live with Docker, Forgejo and a reverse proxy, plus a security mistake I found and closed myself.",{},3,"\u002Fwerk\u002Fself-hosting",{"title":174,"description":253},"werk\u002Fself-hosting","P_LCi3DqS23ZqMg6-W2_qxzE0U-3k5H2Kw4468Qzjn8",1794308981352]