Core Java

Runtime Security in Java: Input Validation, Sandboxing, Safe Deserialization

Your Java application just got pwned. An attacker sent a crafted JSON payload, your deserialization code helpfully executed it, and now they’re downloading your customer database. This isn’t a hypothetical scenario—it’s happened to Equifax, Apache, and countless others.

Runtime security isn’t about firewalls or authentication. It’s about what happens after untrusted data enters your application. Can an attacker trick your code into doing something you never intended? The answer is usually yes, unless you’ve deliberately made it hard.

Java gives you tools to defend yourself. Most developers ignore them because they seem paranoid or overcomplicated. Then production gets compromised, and suddenly those “paranoid” measures seem pretty reasonable.

Why Runtime Security Gets Ignored

You’re focused on features. Security reviews happen later, if at all. The code works in testing, so it ships. Then someone finds your public API accepts user input without validation, or discovers you’re deserializing untrusted data, or realizes your plugin system runs third-party code with full privileges.

The problem is that most vulnerabilities don’t look dangerous when you write them. A simple ObjectInputStream.readObject() call seems innocent until someone explains how it enables remote code execution. Skipping input validation saves five minutes of development time but costs you a security incident six months later.

Security isn’t sexy, it doesn’t show up in demos, and it’s hard to quantify until something breaks. But runtime security issues are among the most exploited vulnerabilities in production systems. Let’s talk about the big three: input validation, sandboxing, and deserialization.

Input Validation: Trust Nothing

Every piece of data entering your application from the outside is a potential attack vector. User input, API requests, file uploads, database records from shared databases, configuration files—all of it.

The rule is simple: validate everything at the boundary. Don’t validate later in your business logic. Don’t assume the frontend validated it. Validate when data enters your system.

What Bad Validation Looks Like

Here’s code I see constantly in production:

@PostMapping("/users")
public ResponseEntity<User> createUser(@RequestBody UserRequest request) {
    User user = new User();
    user.setEmail(request.getEmail());
    user.setAge(request.getAge());
    user.setRole(request.getRole());
    
    userRepository.save(user);
    return ResponseEntity.ok(user);
}

Looks fine, right? It’s a disaster. An attacker can send:

  • Email: "admin@evil.com<script>alert('xss')</script>"
  • Age: -1 or 999999
  • Role: "ADMIN" (escalating their own privileges)

Your application happily accepts all of it because you trusted the input.

Proper Input Validation

Here’s how you actually do it:

public class UserRequest {
    @NotNull(message = "Email is required")
    @Email(message = "Must be a valid email")
    @Size(max = 255, message = "Email too long")
    private String email;
    
    @NotNull(message = "Age is required")
    @Min(value = 0, message = "Age must be positive")
    @Max(value = 150, message = "Age unrealistic")
    private Integer age;
    
    @NotNull(message = "Role is required")
    @Pattern(regexp = "^(USER|MODERATOR)$", message = "Invalid role")
    private String role;
}

@PostMapping("/users")
public ResponseEntity<User> createUser(@Valid @RequestBody UserRequest request) {
    // If validation fails, Spring automatically returns 400 Bad Request
    
    User user = new User();
    user.setEmail(sanitizeEmail(request.getEmail()));
    user.setAge(request.getAge());
    user.setRole(request.getRole());
    
    userRepository.save(user);
    return ResponseEntity.ok(user);
}

private String sanitizeEmail(String email) {
    // Additional layer: remove any HTML/script tags just in case
    return email.replaceAll("<[^>]*>", "");
}

Notice the layered approach. Bean Validation annotations catch obvious problems. Then you sanitize inputs even after validation. This defense-in-depth approach means even if one layer fails, you’re still protected.

Validating Complex Objects

Real applications deal with nested objects, lists, and complex structures:

public class OrderRequest {
    @NotNull
    @Valid  // This is crucial - validates nested objects
    private Customer customer;
    
    @NotEmpty(message = "Order must contain items")
    @Size(max = 100, message = "Too many items")
    @Valid
    private List items;
    
    @NotNull
    @DecimalMin(value = "0.01", message = "Total must be positive")
    private BigDecimal total;
}

public class OrderItem {
    @NotBlank
    @Size(max = 50)
    private String productId;
    
    @Min(1)
    @Max(999)
    private Integer quantity;
    
    @DecimalMin("0.01")
    private BigDecimal price;
}

The @Valid annotation on nested objects is easy to forget but critical. Without it, nested objects bypass validation entirely.

Custom Validators for Business Rules

Sometimes bean validation isn’t enough. You need business logic:

@Target({ElementType.FIELD})
@Retention(RetentionPolicy.RUNTIME)
@Constraint(validatedBy = SafeFilenameValidator.class)
public @interface SafeFilename {
    String message() default "Unsafe filename";
    Class<?>[] groups() default {};
    Class<? extends Payload>[] payload() default {};
}

public class SafeFilenameValidator implements ConstraintValidator<SafeFilename, String> {
    private static final Pattern DANGEROUS_PATTERNS = Pattern.compile(
        "(\\.\\./|\\.\\.\\\\|[<>:\"|?*]|^\\.|\\.$)"
    );
    
    @Override
    public boolean isValid(String filename, ConstraintValidatorContext context) {
        if (filename == null) {
            return true; // Use @NotNull separately
        }
        
        // Prevent path traversal attacks
        if (DANGEROUS_PATTERNS.matcher(filename).find()) {
            return false;
        }
        
        // Whitelist approach: only allow safe characters
        if (!filename.matches("^[a-zA-Z0-9_.-]+$")) {
            return false;
        }
        
        return true;
    }
}

Now you can use @SafeFilename on any file upload parameter. This catches path traversal attacks where an attacker tries uploading to ../../../etc/passwd.

The Whitelist vs Blacklist Trap

When validating input, developers often try to block “bad” characters. This is blacklisting, and it’s almost always wrong:

// BAD: Blacklist approach
public boolean isValidUsername(String username) {
    return !username.contains("<") && 
           !username.contains(">") && 
           !username.contains("'") &&
           !username.contains("\"") &&
           !username.contains("script");
           // You'll never list all dangerous patterns
}

Attackers are creative. They find ways around your blacklist using Unicode characters, URL encoding, double encoding, and tricks you haven’t thought of.

Instead, whitelist what you allow:

// GOOD: Whitelist approach
public boolean isValidUsername(String username) {
    return username.matches("^[a-zA-Z0-9_-]{3,20}$");
    // Only alphanumeric, underscore, hyphen, 3-20 chars
}

If it’s not explicitly allowed, it’s rejected. Much safer.

Sandboxing: Limiting Damage

Input validation prevents bad data from entering. Sandboxing limits what code can do even if an attack succeeds. If your application runs untrusted code—plugins, user scripts, dynamic class loading—sandboxing is essential.

The Java Security Manager (Legacy Approach)

For years, Java used the Security Manager for sandboxing. It’s deprecated as of Java 17 and will be removed, but understanding it helps grasp the concepts:

// Old approach (deprecated)
System.setSecurityManager(new SecurityManager());

// Define permissions in a policy file
grant codeBase "file:/path/to/untrusted/*" {
    permission java.io.FilePermission "/tmp/*", "read,write";
    permission java.net.SocketPermission "example.com:80", "connect";
    // Very limited permissions
};

The Security Manager could restrict what code could do: file access, network access, system property access, etc. It was powerful but complex and had performance overhead.

Modern Sandboxing Approaches

Without Security Manager, you need alternative strategies.

Isolate in separate processes. The most reliable sandbox is a process boundary:

public class PluginExecutor {
    public String executePlugin(String pluginPath, String input) throws Exception {
        ProcessBuilder pb = new ProcessBuilder(
            "java",
            "-Xmx256m",  // Limit memory
            "-classpath", pluginPath,
            "com.example.PluginRunner",
            input
        );
        
        // Restrict what the process can do
        pb.environment().clear();  // No environment variables
        pb.directory(new File("/tmp/sandbox"));  // Restricted directory
        
        Process process = pb.start();
        
        // Timeout protection
        if (!process.waitFor(10, TimeUnit.SECONDS)) {
            process.destroyForcibly();
            throw new TimeoutException("Plugin execution timeout");
        }
        
        return new String(process.getInputStream().readAllBytes());
    }
}

The plugin runs in its own process with limited resources. If it crashes or misbehaves, your main application isn’t affected. You can use containers or VMs for even stronger isolation.

Use custom ClassLoaders with restrictions:

public class SandboxedClassLoader extends ClassLoader {
    private final Set<String> allowedPackages;
    
    public SandboxedClassLoader(Set<String> allowedPackages) {
        super(SandboxedClassLoader.class.getClassLoader());
        this.allowedPackages = allowedPackages;
    }
    
    @Override
    protected Class<?> loadClass(String name, boolean resolve) 
            throws ClassNotFoundException {
        // Block dangerous classes
        if (name.startsWith("java.lang.Runtime") ||
            name.startsWith("java.lang.ProcessBuilder") ||
            name.startsWith("sun.misc.Unsafe")) {
            throw new ClassNotFoundException("Access denied: " + name);
        }
        
        // Whitelist specific packages only
        boolean allowed = allowedPackages.stream()
            .anyMatch(name::startsWith);
            
        if (!allowed) {
            throw new ClassNotFoundException("Package not whitelisted: " + name);
        }
        
        return super.loadClass(name, resolve);
    }
}

// Usage
Set<String> allowed = Set.of("com.example.safe.", "org.apache.commons.lang3.");
ClassLoader sandboxed = new SandboxedClassLoader(allowed);
Class<?> pluginClass = sandboxed.loadClass("com.example.safe.UserPlugin");

This prevents plugins from loading dangerous classes. It’s not bulletproof—determined attackers might find reflection-based workarounds—but it raises the bar significantly.

Limit resource consumption:

public class ResourceLimitedExecutor {
    private final ExecutorService executor = Executors.newFixedThreadPool(4);
    
    public <T> T executeWithLimits(Callable<T> task, 
                                   long timeoutSeconds,
                                   long maxMemoryMB) throws Exception {
        // CPU/time limit via timeout
        Future<T> future = executor.submit(task);
        
        try {
            return future.get(timeoutSeconds, TimeUnit.SECONDS);
        } catch (TimeoutException e) {
            future.cancel(true);
            throw new RuntimeException("Task exceeded time limit");
        }
        
        // Memory limits are harder - best handled at JVM level with -Xmx
        // Or use process isolation as shown earlier
    }
}

Even untrusted code can’t consume infinite CPU if you enforce timeouts. Memory is trickier—process isolation or container limits work better than trying to enforce it in-JVM.

Real-World Sandboxing Example

Let’s say you’re building a system that runs user-submitted data transformation scripts:

public class ScriptSandbox {
    private static final long MAX_EXECUTION_TIME_MS = 5000;
    private static final String SANDBOX_DIR = "/tmp/script-sandbox";
    
    public String executeScript(String script, String data) {
        // 1. Validate the script isn't obviously malicious
        if (containsDangerousPatterns(script)) {
            throw new SecurityException("Script contains forbidden patterns");
        }
        
        // 2. Write script to isolated directory
        Path scriptPath = Paths.get(SANDBOX_DIR, UUID.randomUUID().toString() + ".js");
        Files.writeString(scriptPath, script);
        
        try {
            // 3. Execute in separate process with resource limits
            ProcessBuilder pb = new ProcessBuilder(
                "timeout", String.valueOf(MAX_EXECUTION_TIME_MS / 1000),
                "node",
                "--max-old-space-size=100",  // 100MB memory limit
                scriptPath.toString()
            );
            
            pb.directory(new File(SANDBOX_DIR));
            pb.redirectErrorStream(true);
            
            Process process = pb.start();
            
            // 4. Pass data via stdin, read result from stdout
            try (OutputStream os = process.getOutputStream()) {
                os.write(data.getBytes());
            }
            
            String result = new String(process.getInputStream().readAllBytes());
            
            int exitCode = process.waitFor();
            if (exitCode != 0) {
                throw new RuntimeException("Script failed with exit code: " + exitCode);
            }
            
            return result;
            
        } finally {
            // 5. Clean up
            Files.deleteIfExists(scriptPath);
        }
    }
    
    private boolean containsDangerousPatterns(String script) {
        // Check for obvious attacks
        return script.contains("require('child_process')") ||
               script.contains("eval(") ||
               script.contains("Function(") ||
               script.matches(".*\\brequire\\s*\\(.*");
    }
}

This example combines multiple defenses: static analysis, process isolation, resource limits, and cleanup. No single defense is perfect, but layers make exploitation much harder.

Safe Deserialization: The Biggest Landmine

Java deserialization vulnerabilities are responsible for some of the worst security breaches in history. The problem is fundamental: deserialization can execute arbitrary code during object construction.

Why Deserialization Is Dangerous

When you deserialize an object, Java calls constructors, readObject methods, and other code. An attacker who controls serialized data can craft objects that execute arbitrary commands:

// DANGEROUS CODE - DO NOT USE IN PRODUCTION
public void loadUserSettings(byte[] data) {
    try (ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(data))) {
        UserSettings settings = (UserSettings) ois.readObject();
        applySettings(settings);
    }
}

This looks innocent. But an attacker can send serialized data containing objects from libraries on your classpath (like Apache Commons Collections) that execute system commands during deserialization. They never even need to touch your UserSettings class.

The infamous “gadget chains” exploit this. By chaining together standard library classes in specific ways, attackers achieve remote code execution. Tools like ysoserial automate creating these payloads.

Never Deserialize Untrusted Data

The safest approach is simple: don’t use Java serialization for data from untrusted sources. Period.

Use JSON, Protocol Buffers, or other data-only formats instead:

// SAFE: Using JSON
public UserSettings loadUserSettings(String json) {
    ObjectMapper mapper = new ObjectMapper();
    return mapper.readValue(json, UserSettings.class);
}

JSON parsers like Jackson don’t execute arbitrary code during parsing. They just populate fields. The attack surface shrinks dramatically.

When You Must Deserialize

Sometimes you’re stuck with Java serialization—legacy protocols, caching libraries, or distributed computing frameworks. If you absolutely must deserialize untrusted data, use defensive measures.

Use ObjectInputFilter (Java 9+):

public Object safeDeserialize(byte[] data) throws Exception {
    try (ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(data))) {
        // Whitelist allowed classes
        ObjectInputFilter filter = ObjectInputFilter.Config.createFilter(
            "com.example.UserSettings;" +
            "com.example.UserPreference;" +
            "java.util.ArrayList;" +
            "java.lang.String;" +
            "!*"  // Reject everything else
        );
        
        ois.setObjectInputFilter(filter);
        
        return ois.readObject();
    }
}

The filter explicitly whitelists safe classes and rejects everything else. This blocks gadget chains that rely on unexpected classes being available.

Validate object graphs:

public class SafeObjectInputStream extends ObjectInputStream {
    private final Set<String> allowedClasses;
    private int maxDepth = 10;
    private int currentDepth = 0;
    
    public SafeObjectInputStream(InputStream in, Set<String> allowedClasses) 
            throws IOException {
        super(in);
        this.allowedClasses = allowedClasses;
    }
    
    @Override
    protected Class<?> resolveClass(ObjectStreamClass desc) 
            throws IOException, ClassNotFoundException {
        // Check depth to prevent deeply nested objects
        if (++currentDepth > maxDepth) {
            throw new InvalidClassException("Max depth exceeded");
        }
        
        String className = desc.getName();
        
        // Whitelist check
        if (!allowedClasses.contains(className)) {
            throw new InvalidClassException("Class not allowed: " + className);
        }
        
        return super.resolveClass(desc);
    }
    
    @Override
    protected ObjectStreamClass readClassDescriptor() 
            throws IOException, ClassNotFoundException {
        ObjectStreamClass desc = super.readClassDescriptor();
        currentDepth--;
        return desc;
    }
}

This custom implementation adds another layer of defense by tracking deserialization depth and enforcing strict whitelisting.

Sign serialized data:

public class SignedSerializer {
    private final SecretKey signingKey;
    
    public byte[] serialize(Object obj) throws Exception {
        // Serialize object
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        try (ObjectOutputStream oos = new ObjectOutputStream(baos)) {
            oos.writeObject(obj);
        }
        byte[] data = baos.toByteArray();
        
        // Create signature
        Mac mac = Mac.getInstance("HmacSHA256");
        mac.init(signingKey);
        byte[] signature = mac.doFinal(data);
        
        // Combine signature and data
        ByteBuffer buffer = ByteBuffer.allocate(signature.length + data.length);
        buffer.put(signature);
        buffer.put(data);
        
        return buffer.array();
    }
    
    public Object deserialize(byte[] signedData) throws Exception {
        ByteBuffer buffer = ByteBuffer.wrap(signedData);
        
        // Extract signature and data
        byte[] signature = new byte[32];  // HmacSHA256 produces 32 bytes
        buffer.get(signature);
        
        byte[] data = new byte[buffer.remaining()];
        buffer.get(data);
        
        // Verify signature
        Mac mac = Mac.getInstance("HmacSHA256");
        mac.init(signingKey);
        byte[] expectedSignature = mac.doFinal(data);
        
        if (!MessageDigest.isEqual(signature, expectedSignature)) {
            throw new SecurityException("Signature verification failed");
        }
        
        // Deserialize if signature valid
        try (ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(data))) {
            return ois.readObject();
        }
    }
}

Signing prevents attackers from tampering with serialized data. They can’t inject malicious objects without the signing key. This works when data might be exposed but not directly controlled by attackers (like client-side storage or caching systems).

Alternative Serialization Libraries

Several libraries provide safer serialization:

Kryo offers better performance and can be configured to whitelist classes:

Kryo kryo = new Kryo();
kryo.setRegistrationRequired(true);  // Reject unregistered classes
kryo.register(UserSettings.class);
kryo.register(ArrayList.class);

// Serialize
Output output = new Output(new FileOutputStream("file.bin"));
kryo.writeObject(output, userSettings);
output.close();

// Deserialize - only registered classes allowed
Input input = new Input(new FileInputStream("file.bin"));
UserSettings settings = kryo.readObject(input, UserSettings.class);
input.close();

Protocol Buffers or Apache Avro use schema-based serialization. They’re verbose to set up but completely avoid code execution risks:

message UserSettings {
  string theme = 1;
  int32 fontSize = 2;
  repeated string favorites = 3;
}

These formats only deserialize data, never code. It’s impossible to achieve code execution through protobuf deserialization.

Real-World Security Incident: A Cautionary Tale

A company I consulted for had an admin portal that accepted file uploads for batch processing. The code looked like this:

@PostMapping("/admin/import")
public String importData(@RequestParam("file") MultipartFile file) {
    try {
        byte[] data = file.getBytes();
        ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(data));
        DataImport importData = (DataImport) ois.readObject();
        
        processImport(importData);
        return "Import successful";
    } catch (Exception e) {
        return "Import failed: " + e.getMessage();
    }
}

The developers thought it was safe because the endpoint required admin authentication. What they missed:

  1. An attacker compromised a low-level admin account through phishing
  2. The attacker uploaded a malicious serialized payload using ysoserial
  3. During deserialization, the payload executed system commands
  4. The attacker gained shell access to the application server
  5. From there, they pivoted to the database and exfiltrated customer data

The fix required multiple changes:

@PostMapping("/admin/import")
public String importData(@RequestParam("file") MultipartFile file) {
    // Validate file type
    if (!file.getContentType().equals("application/json")) {
        return "Only JSON imports allowed";
    }
    
    // Validate file size
    if (file.getSize() > 10 * 1024 * 1024) {  // 10MB limit
        return "File too large";
    }
    
    try {
        // Use JSON instead of Java serialization
        ObjectMapper mapper = new ObjectMapper();
        mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
        
        DataImport importData = mapper.readValue(
            file.getInputStream(), 
            DataImport.class
        );
        
        // Validate imported data
        validateImportData(importData);
        
        // Process in restricted context
        processImportSafely(importData);
        
        return "Import successful";
    } catch (Exception e) {
        log.error("Import failed", e);
        return "Import failed - check logs";
    }
}

The incident cost them millions in incident response, legal fees, and reputation damage. All because of one unsafe deserialization call.

Practical Security Checklist

Here’s what you should be doing in every Java application:

Input validation:

  • Use Bean Validation annotations on all DTOs
  • Validate nested objects with @Valid
  • Whitelist acceptable patterns, don’t blacklist dangerous ones
  • Sanitize data even after validation
  • Validate file uploads: type, size, content
  • Never trust client-side validation alone

Sandboxing:

  • Run untrusted code in separate processes or containers
  • Use custom ClassLoaders to restrict class access
  • Enforce resource limits: memory, CPU time, disk space
  • Clean up temporary files and resources
  • Log all sandbox violations

Deserialization:

  • Prefer JSON/Protocol Buffers over Java serialization
  • Never deserialize untrusted data without filters
  • Use ObjectInputFilter to whitelist classes
  • Sign serialized data when possible
  • Regularly audit classpath dependencies for known gadgets
  • Consider Kryo with registration-required mode

General practices:

  • Keep dependencies updated (exploits target specific versions)
  • Use static analysis tools to catch security issues
  • Log security-relevant events for monitoring
  • Test with malicious inputs, not just happy paths
  • Assume everything can be attacked

Tools That Help

SpotBugs with FindSecBugs plugin catches common security issues at build time:

<plugin>
    <groupId>com.github.spotbugs</groupId>
    <artifactId>spotbugs-maven-plugin</artifactId>
    <configuration>
        <plugins>
            <plugin>
                <groupId>com.h3xstream.findsecbugs</groupId>
                <artifactId>findsecbugs-plugin</artifactId>
                <version>1.12.0</version>
            </plugin>
        </plugins>
    </configuration>
</plugin>

OWASP Dependency-Check identifies vulnerable dependencies:

<plugin>
    <groupId>org.owasp</groupId>
    <artifactId>dependency-check-maven</artifactId>
    <executions>
        <execution>
            <goals>
                <goal>check</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Snyk or Dependabot automate dependency updates when vulnerabilities are disclosed.

The Mindset Shift

Security isn’t a feature you add at the end. It’s a constraint you design for from the start. Every time you accept external input, ask yourself: “What’s the worst an attacker could do with this?” Every time you deserialize data, ask: “Do I trust this data’s source completely?”

Paranoia in code review is a virtue. When someone’s PR includes deserialization or dynamic class loading, question it aggressively. When input validation is missing, send it back. Better to seem pedantic in code review than negligent after a breach.

Runtime security is about reducing trust. Don’t trust user input. Don’t trust plugins. Don’t trust serialized data. Don’t trust that your validation is perfect. Layer defenses so that when one fails—and it will—others catch the attack.

The good news is that once you internalize these patterns, they become second nature. Input validation becomes automatic. You instinctively avoid Java serialization. You design with isolation in mind. Security becomes part of your coding style, not something bolted on later.

Useful Resources

Eleftheria Drosopoulou

Eleftheria is an Experienced Business Analyst with a robust background in the computer software industry. Proficient in Computer Software Training, Digital Marketing, HTML Scripting, and Microsoft Office, they bring a wealth of technical skills to the table. Additionally, she has a love for writing articles on various tech subjects, showcasing a talent for translating complex concepts into accessible content.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Back to top button