CoreWeave launches one-click GPT-J inference service with scale-to-zero
CoreWeave says it is making EleutherAI’s GPT-J available through a one-click inference service, combining the open-source model with access to its GPU infrastructure. The company is targeting organizations that want to deploy GPT-J with faster scaling and lower infrastructure overhead.
- CoreWeave offers one-click deployment for GPT-J inference.
- GPT-J has 6B parameters and was trained on The Pile.
- The service supports scale-to-zero with zero resource consumption when scaled down.
- AI Dungeon reported 75% lower costs and 50% lower latency.
- NovelAI reported 3x faster requests and 75% lower cloud costs.
A smaller model designed for practical deployment
GPT-J was trained by EleutherAI on The Pile, described by CoreWeave as an 825 GiB language modelling data set. The company says GPT-J is one of the largest open-source natural language processing models available, with 6B parameters.
CoreWeave says the model can compete with, and at times produce higher quality results than, GPT-3 and its 175B parameters. Its smaller size means GPT-J requires less memory and horsepower to serve at scale, according to the company.
EleutherAI said its aim was to create “the world's largest open-source language model” and described diverse data as important for improving the quality of large models. The organization said GPT-J-6B is “proving to be incredibly powerful given its size and quality.”
One-click deployment and responsive scaling
CoreWeave’s GPT-J Inference Service is available through CoreWeave Apps, where clients can deploy an API inference service with one click. CoreWeave says this removes infrastructure overhead while allowing customers to use its range of GPUs, including NVIDIA A5000s.
The service uses CoreWeave’s Kubernetes-native infrastructure and supports scale-to-zero. When workloads are scaled down, the service consumes zero resources and incurs zero billing, according to CoreWeave. The company says its autoscaling is responsive and that GPT-J can be served more efficiently by selecting from a broad range of GPUs.
CoreWeave also says its fleet includes approximately 50,000 GPUs. Customers can deploy containerized workloads through Kubernetes, while CoreWeave engineers provide help with DevOps, workload optimization and GPU selection for training and serving models.
Early customer results and open-source NLP support
AI Dungeon CEO Nick Walton said the company previously faced planning, cost and latency problems with GPT-3. After moving its compute infrastructure to CoreWeave, AI Dungeon lowered costs by 75% and reduced latency by 50%, according to CoreWeave. The company expects further performance gains as it moves production traffic to GPT-J on CoreWeave.
AI Dungeon selected the A5000 after benchmarking CoreWeave’s available compute and said it delivered the best performance-adjusted cost for its workloads. NovelAI is also serving its Sigurd storytelling model, which is based on GPT-J and fine tuned on novels and short stories. NovelAI CEO Eren Doğan said requests were served 3x faster after the move to CoreWeave, while cloud costs fell by 75%.
CoreWeave says it is working with EleutherAI on open-source NLP using a DGX A100 distributed training cluster with full InfiniBand connectivity between nodes. For users and developers, the announcement offers a managed route to deploy GPT-J, with the company’s reported cost and scaling benefits aimed at making production NLP workloads easier to operate.
CoreWeave: CoreWeave Powers a World of Possibility with GPT-J


