When using Hibernate for saving a large number of entities to database, You can meet an OutOfMemoryException. The tutorial will show the case and resolve it with Hibernate Batch processing.
Related articles:
– How to start development with Hibernate – XML Mapping File Hibernate
I. Technologies
– Java 1.8
– Maven 3.3.9
– Spring Tool Suite – Version 3.8.1.RELEASE
– Hibernate: 5.2.10.Final
– MySql database
II. Problem & Solution
1. Problem
Using Hibernate to save 20000 entities to database as below code:
Session session = factory.openSession();
Transaction tx = null;
try {
tx = session.beginTransaction();
for (int i = 0; i < 20000; i++) {
String text = Utilities.generatedRandomString();
Data data = new Data(text);
session.save(data);
}
tx.commit();
} catch (Exception e) {
if (null != tx) {
tx.rollback();
}
} finally {
session.close();
}
An Exception java.lang.OutOfMemoryError will be thrown somewhere around the 16700th records:
... Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.jar.Attributes.read(Attributes.java:394) ...
- The same issue when using Hibernate to update 20000 entities with below code:
Session session = factory.openSession();
Transaction tx = null;
try {
tx = session.beginTransaction();
ScrollableResults dataCursor = session.createQuery("FROM Data").scroll();
while (dataCursor.next()) {
Data data = (Data) dataCursor.get(0);
String newText = Utilities.generatedRandomString();
data.setText(newText);
session.update(data);
}
tx.commit();
} catch (Exception e) {
if (null != tx) {
tx.rollback();
}
} finally {
session.close();
}
Why ?
Because Hibernate has a mandatory cache is Session, and all the persisted objects will be stored in Session before commit() function is called. So OutOfMemoryException will be thrown out when Session cache meets the limit of Heap size.
2. Solution
Hibernate Batch processing will resolve the problem.
Step to configure Batch processing:
2.1 Config Batch size
- Set hibernate.jdbc.batch_size properties in hibernate.cfg.xml.
hibernate.jdbc.batch_size should be range from 10 to 50 (an reasonable number should be chose base on Object's size).
- Hibernate uses hibernate.jdbc.batch_size to determine the number of persisted objects will be inserted as batch then release it from caching.
2.2 Modify Batch processing code
- With Batch Insert processing, modify code as below:
Session session = factory.openSession();
Transaction tx = null;
try {
tx = session.beginTransaction();
for (int i = 0; i < 20000; i++) {
String text = Utilities.generatedRandomString();
Data data = new Data(text);
session.save(data);
if (i % 50 == 0) {
System.out.println("==========================================log: i = " + i);
session.flush();
session.clear();
}
}
tx.commit();
} catch (Exception e) {
if (null != tx) {
tx.rollback();
}
} finally {
session.close();
}
- With Batch Updates processing, modify code as below:
Session session = factory.openSession();
Transaction tx = null;
try {
tx = session.beginTransaction();
ScrollableResults dataCursor = session.createQuery("FROM Data").scroll();
int count = 1;
while (dataCursor.next()) {
Data data = (Data) dataCursor.get(0);
String newText = Utilities.generatedRandomString();
data.setText(newText);
session.update(data);
if (count % 50 == 0) {
System.out.println("============================log: count = " + count);
session.flush();
session.clear();
}
count++;
}
tx.commit();
} catch (Exception e) {
if (null != tx) {
tx.rollback();
}
} finally {
session.close();
}
III. Practices
Step to do:
– Create Spring Maven project
– Create persistent class
– Setup Hibernate configuration
– Implement Hibernate Application
– Run & Check Results
1. Create Spring Maven project
Using Spring Tool Suite, create a Simple Spring Maven project.
Then modify pom.xml file with Hibernate & MySql dependencies:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.springframework.samples</groupId>
<artifactId>SpringHibernateStarter</artifactId>
<version>0.0.1</version>
<dependencies>
<!-- Hibernate -->
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>5.2.10.Final</version>
</dependency>
<!-- MySQL connector -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>6.0.6</version>
</dependency>
</dependencies>
</project>
2. Create persistent class
- Create a Data.java:
package com.javasampleapproach.hibernate.model;
public class Data {
private int id;
private String text;
public Data(){};
public Data(String text){
this.text = text;
}
public void setId(int id){
this.id = id;
}
public int getId(){
return this.id;
}
public void setText(String text){
this.text = text;
}
public String getText(){
return this.text;
}
}
3. Setup Hibernate configuration
Under folder /src/main/resources:
– Create Hibernate Configuration file – hibernate.cfg.xml:
Define hibernate.jdbc.batch_size = 50
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/testdb</property>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.connection.password"></property>
<property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="show_sql">true</property>
<property name="hibernate.jdbc.batch_size">50</property>
<mapping resource="Data.hbm.xml"></mapping>
</session-factory>
</hibernate-configuration>
– Create a mapping file for Data class – Data.hbm.xml:
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE hibernate-mapping PUBLIC
"-//Hibernate/Hibernate Mapping DTD//EN"
"http://www.hibernate.org/dtd/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
<class name="com.javasampleapproach.hibernate.model.Data" table="data">
<meta attribute="class-description">
Mapping between persistent class - Data with MySQL table - data
</meta>
<id name="id" type="int" column="id">
<generator class="native"/>
</id>
<property name="text" column="text" type="string"/>
</class>
</hibernate-mapping>
4. Implement Hibernate Application
- Firstly, create a Utilities class has a utility function: generatedRandomString()
package com.javasampleapproach.hibernate;
import org.apache.commons.lang.RandomStringUtils;
public class Utilities {
public static String generatedRandomString(){
return RandomStringUtils.random(50000, true, true);
}
}
– In Application.java, implement 4 functions:
setUpSessionFactory(): setup Hibernate Session Factory.
storeData(): save Data entities to database.
updateData(): update Data entities.
shutdown(): close Hiberbate Session Factory Object.
package com.javasampleapproach.hibernate;
import org.hibernate.ScrollableResults;
import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.hibernate.Transaction;
import org.hibernate.cfg.Configuration;
import com.javasampleapproach.hibernate.model.Data;
public class Application {
private static SessionFactory factory = null;
public static void main(String[] args) {
System.out.println("============Set up Session Factory============");
setUpSessionFactory();
System.out.println("============Store Data============");
storeData();
/*System.out.println("============Update Data============");
updateData();
*/
shutdown();
}
public static void setUpSessionFactory() {
// create sessionFactory
try {
factory = new Configuration().configure().buildSessionFactory();
} catch (Throwable ex) {
System.err.println("Failed to create sessionFactory object." + ex);
throw new ExceptionInInitializerError(ex);
}
}
public static void shutdown() {
factory.close();
}
public static void storeData() {
Session session = factory.openSession();
Transaction tx = null;
try {
tx = session.beginTransaction();
for (int i = 0; i < 20000; i++) {
String text = Utilities.generatedRandomString();
Data data = new Data(text);
session.save(data);
if (i % 50 == 0) {
System.out.println("==========================================i = " + i);
session.flush();
session.clear();
}
}
tx.commit();
} catch (Exception e) {
if (null != tx) {
tx.rollback();
}
} finally {
session.close();
}
}
public static void updateData() {
Session session = factory.openSession();
Transaction tx = null;
try {
tx = session.beginTransaction();
ScrollableResults dataCursor = session.createQuery("FROM Data").scroll();
int count = 1;
while (dataCursor.next()) {
Data data = (Data) dataCursor.get(0);
String newText = Utilities.generatedRandomString();
data.setText(newText);
session.update(data);
if (count % 50 == 0) {
System.out.println("==========================================count = " + count);
session.flush();
session.clear();
}
count++;
}
tx.commit();
} catch (Exception e) {
if (null != tx) {
tx.rollback();
}
} finally {
session.close();
}
}
}
5. Run & Check Results
– Create a MySql data table by script:
CREATE TABLE data( id INT NOT NULL AUTO_INCREMENT, text VARCHAR(50000) NOT NULL, PRIMARY KEY (id) );
5.1 Batch Insert
- Run the class Application.java with main modified function:
public static void main(String[] args) {
setUpSessionFactory();
storeData();
shutdown();
}
Insert successfully with Heap Memory Usage chart:
5.2 Batch Update
- Run the class Application.java with main modified function:
public static void main(String[] args) {
setUpSessionFactory();
updateData();
shutdown();
}
Insert successfully with Heap Memory Usage chart:
