Can we summarize at scale?

AI tools like GPT-3 have strict token or character limits to the text you can input, because the models are resource intensive to run. Of course GPT-3 is already good enough to not need a lot of context to complete tasks, but sometimes the limit is overly restrictive. For example if the task involved is to summarize a large block of text, say a book or anything over 1,500 words, you’re likely to run into their 4,000 token limit (prompt + completion). This can be a problem because many of the tasks we want to use GPT-3 for require multiple lines of text, or multiple examples fed to the AI so it knows what to do. The costs are negligible relative to the value of what you can create, so many people were interested in a workaround, where they could tradeoff more computing cost for longer text inputs.

To find a workaround, the OpenAI team experimented with progressive summarization, outlined in a paper they published (though they didn’t release the code or model publicly). The way it worked was quite elegant: break the large text into smaller chunks or sections, then summarize each section of text, before finally summarizing the summaries. The end result gives us a surprisingly good summary of all the text underneath. It still needs to be checked by a human for accuracy and small edits, but it saves a significant amount of time by letting you dig into interesting texts based on their summaries, rather than having to waste time reading the whole thing first. Editing is always a much cheaper and more efficient task than writing.