Technology Research & Proof-of-Concept

• Research and evaluate new technology stacks, databases, storage systems, distributed systems, and infrastructure solutions to address existing technical challenges and future team requirements.
• Build Proof-of-Concept implementations to evaluate technologies before production adoption.
• Evaluate performance, scalability, reliability, compatibility, and operational complexity.
• Design benchmarks and workload tests to validate technical hypotheses and compare alternative solutions.
• Analyze technical trade-offs and recommend solutions suitable for actual workloads and operational conditions.
  

In-house Storage & Database Systems

• Develop, maintain, and operate in-house database and storage clusters used for long-term file, photo, and video storage for Zalo product features.
• Analyze and resolve issues related to system performance, scalability, reliability, and capacity.
• Implement system changes, upgrades, and optimizations required to support data and workload growth.
• Develop and improve internal tooling and automation for deployment, operation, monitoring, and troubleshooting.
• Investigate complex bottlenecks and issues related to database and storage workloads.

Third-party Distributed Systems

• Operate, maintain, troubleshoot, and tune systems such as ScyllaDB, SeaweedFS, SQL databases, Kafka, and other database, storage, or messaging platforms used by the team.
• Perform cluster deployment, upgrades, configuration changes, scaling, and capacity expansion.
• Monitor production system health, performance, and resource utilization.
• Investigate issues related to latency, throughput, availability, and resource usage.
• Perform root cause analysis for complex issues and propose solutions to prevent recurrence.

Linux, Hardware & Performance Engineering

• Research, operate, troubleshoot, and optimize Linux-based systems, primarily CentOS 7 and Rocky Linux 9.
• Investigate operating system issues affecting production stability, latency, throughput, or resource utilization.
• Analyze performance across multiple layers, from applications and database/storage services to operating systems and hardware, to identify bottlenecks and root causes.
• Tune Linux systems according to workload characteristics, including CPU/scheduling, memory/page cache, network stack, filesystems, disk I/O, resource limits, and kernel parameters.
• Investigate memory pressure, resource contention, I/O latency, network performance, and filesystem behavior.
• Perform benchmarks and workload tests to evaluate the impact of operating system or hardware configuration changes.
• Research and evaluate hardware for database and storage workloads, including RAM, NICs, RAID controllers/configurations, SSDs, NVMe devices, HDDs, and server configurations affecting performance or stability.
• Compare hardware and configuration alternatives based on actual workloads and benchmark results.
• Recommend configurations by considering performance, reliability, capacity, and cost trade-offs.
  

Monitoring, Reliability & Troubleshooting

• Build and improve monitoring, metrics, logging, and alerting for assigned systems.
• Monitor performance, resource utilization, capacity, and overall system health.
• Troubleshoot complex production issues involving applications, databases, operating systems, networks, or hardware.
• Perform root cause analysis and recommend corrective actions for significant incidents. 
• Improve tooling, automation, and configuration to increase system reliability and reduce operational workload.

Cross-team Coordination

• Coordinate with product teams, infrastructure teams, and other engineering teams to resolve dependencies, support requirements, and technical issues during implementation. 
• Proactively track and follow up on infrastructure and cross-team dependencies to support product delivery timelines. 
• Collaborate with relevant teams to investigate and resolve issues during integration, deployment, and production operation. 
• Keep track of infrastructure and related system changes that may affect the systems under the team's responsibility.