Heart Rate and Oxygen Saturation Based on Contact Photoplethysmogram

Article Study Case UPGRADE ENTRY LEVEL AND MID RANGE TYPE SMARTPHONE CAMERA TO CALCULATE HEART RATE AND OXYGEN SATURATION BASED ON CONTACT PHOTOPLETHYSMOGRAM Abstract The use of consumer smartphones as medical sensor devices shows great potential, especially with the integration of sensors to track biosignals such as voice and pulse. A major challenge lies in the technical limitations of cell phone cameras in meeting medical accuracy standards, particularly for pulse oximetry measurements. This study introduces an innovative smartphone application capable of estimating vital signs such as heart rate and SpO2 levels. The app applies camera sampling rate enhancement techniques to improve measurement accuracy. In this study, videos of 30 subjects were taken of their fingertips to extract two types of PPG signals, which were then compared based on ensemble average morphology analysis and beat-by-beat evaluation. Results show that a short exposure time produces a CPPG (contact photoplethysmogram) signal with a high SNR of up to 20 dB over the PPG reference signal. The app showed an average error of 1.97 for heart rate measurement and 0.87 for oxygen saturation measurement. On the OPPO A83 smartphone, the average error for heart rate was 2.18 and for oxygen saturation was 0.84. Meanwhile, on the Samsung A12 smartphone, the average error for heart rate was 1.12 and for oxygen saturation was 0.41. These results support the effectiveness of the proposed smart phone application in vital signs estimation. Keywords: entry level, smartphone camera, calculate, heart rate and oxygen saturation, photoplethysmogram INTRODUCTION Cardiovascular disease remains a serious global threat and is the leading cause of death worldwide. According to data from the World Health Organization (WHO), more than 17 million people die each year from heart and vascular diseases. The 2018 Basic Health Research shows a year-on-year increase in heart and vascular disease cases in Indonesia, with approximately 15 in 1000 people or around 2,784,064 individuals suffering from heart disease. Types of cardiovascular disease involve coronary heart disease, heart rhythm disorders, cardiomyopathy, heart valve disease, heart infections, and congenital heart disease. Arrhythmia, as heart rhythm disorder, is a condition where the heart's electrical system is impaired, causing abnormalities in the heart rhythm. AMELIA ROSE SMITH 1 Amelia Rose Smith The effects of arrhythmia include palpitations, dizziness, fainting, stroke, and sudden death. There is a lack of understanding in the community about arrhythmia, it is not as popular as other heart diseases, and there are limited health facilities specialized in arrhythmia as well as a limited number of doctors who are experts in this field. Heart rate monitoring in patients with arrhythmia is very important to know the heart rhythm of a person within a few minutes. Types of arrhythmia include too slow a heart rhythm (bradycardia with a heart rate of less than 60 BPM) and too fast a rhythm (tachycardia with a heart rate of more than 100 BPM). Besides cardiovascular disease, there is also a "silent killer" known as happy hypoxia. Hypoxia is a condition in which oxygen saturation in the blood is very low, characterized by symptoms of shortness of breath and rapid heartbeat (tachycardia). On August 4, 2023, Indonesia announced Presidential Decree No. 48/2023 on the temporary suspension of the handling of the COVID-19 pandemic. The total number of COVID19 cases in Indonesia as of January 20, 2024 was 6,826,664 confirmed cases in 514 districts/cities in 34 provinces. The five provinces that reported the highest number of daily confirmed cases in the third week of 2024 were DKI Jakarta, West Java, East Java, Banten, and Central Java. Especially in the current condition of COVID-19, which has changed its status to endemic, it is necessary to remain vigilant about the phenomenon of "silent hypoxia" or "happy hypoxia," where patients experience severe arterial hypoxia without showing symptoms or signs of respiratory system disorders. Therefore, it is necessary to develop medical technologies that can help doctors, medical personnel, and patients manage these conditions using smartphones. However, entry-level and mid-range smartphone devices, which are widely used by the general public, face a number of obstacles in supporting technologies that can be used to determine heart rate and oxygen saturation. One of the main challenges is the limitation of the cameras on these devices. Although the cameras on modern smartphones have the capability of recording high-definition videos, the quality may still be inadequate for accurate measurement of heart rate (HR) and blood oxygen saturation (SpO2). Research has shown that smartphone cameras can be utilized to record blood flow rate videos and measure HR and SpO2 through digital signal processing (DSP). However, accurate measurements require a high sampling rate, and camera characteristics, such as flashlight brightness level and camera sensitivity, can affect the accuracy of measurements. AMELIA ROSE SMITH 2 Article Study Case In a study conducted by Karlen et al. (2012), they successfully used a smartphone camera to record blood flow rate videos and perform HR and SpO2 measurements with a signal processing-based method. However, integration constraints with medical device ISO standards and variations in camera characteristics are challenges that need to be overcome. Other research results by Tyapochkin (2018) and Rachim et (2019) highlighted that device characteristics and external conditions, such as optical shunts and environmental lighting, can affect accurate measurements. Despite achieving the best possible signal processing and prediction results, the integration of smartphone cameras with the standard ISO requirements of medical devices proved to be challenging. This difficulty mainly stems from the prerogative granted to smartphone manufacturers, which authorizes them to regulate the design and selection of cameras for their respective devices. In addition, recent investigations have revealed that diverse device characteristics and various external conditions, such as possible optical shunts and environmental lighting, may pose potential obstacles in accurate measurement. In particular, variations in camera characteristics, including camera sampling rate, flashlight brightness level, and camera sensitivity, can have a significant impact on measurement accuracy. For example, in conventional devices, the effective analog-to-digital (A/D) conversion rate must be at least three times higher than the LED illumination frequency, which typically exceeds 450 Hz. Unfortunately, many consumer-grade smartphone cameras are limited to a sampling rate of only 25-30 frames per second (fps). This gap has been corroborated by several studies, which achieved high accuracy results using cameras with higher sampling rates. Therefore, the application of this method to cameras with lower sampling rates will inevitably produce variable results. This variance can pose a challenge when investigating pulse rate variability (PRV) and pulse wave propagation throughout the body. This research seeks to create an innovative smartphone physiological sensing system that can accurately estimate heart rate (HR) and blood oxygenation level (SPO2) from contact photoplethysmography (cPPG) measurements using a smartphone camera. New algorithms have been designed and integrated into the system to improve the quality of fingertip video recordings, thereby increasing the reliability and accuracy of the measurements. The following summarizes the main findings of this study: AMELIA ROSE SMITH 3 Amelia Rose Smith A. Proposed and built a smartphone application utilizing signal processing-based methodology to estimate heart rate and SpO2 level. smartphone application to estimate heart rate and spo2 using signal processing-based method. B. Designing an algorithm to increase the sampling rate of the acquired cPPG signal. C. Assessing the performance of our proposed application on various subscriber-level smart phones. The content of this scientific journal consists of four sections. Chapter 2 discusses the technical research methods starting from application design, experimentation, and data collection. Then it continues to display the results and analyze the design and experimental results in chapter 3 while chapter 4 concludes the journal based on the main findings. METHODS A. Research Stages Building a PPG Measurement Smartphone Application Designing Fs enhancement Algorithms for Applications Measure and analyze application performance on various phones Figure 1: Diagram of the Research Phase Mobile app performance experiments were conducted in an indoor laboratory illuminated with LED ambient light. 30 healthy subjects (various ages, finger skin color, skin volume, fat, bone, etc.) aged between 20 and 30 years old, participated in the experiment. Data was collected for 24 subjects using Oppo A83 (single camera) and 6 subjects using Samsung A12 (quad camera). The two predefined smartphones both have different camera features, including design, camera resolution, and aperture but both have similar sampling rates and sequential video resolutions of 30 fps and 1080p. We used these smartphone models to test the impact of various camera configurations. Each participant was welcomed into the experimental room and given information about the purpose and methodology of the study. Their fingertips were placed on the camera lens and illuminated with a flashlight. The fingertip was recorded for 30 seconds with the flashlight on. During the recording, participants were asked to remain still and comfortably place their index finger on the smartphone camera. The participant AMELIA ROSE SMITH 4 Article Study Case inserted his left index finger into the oximeter that served as the benchmark, and the right index finger was directly hooked onto the back camera of the smartphone equipped with a flashlight during the measurement. It took approximately 20 minutes to complete the recording for each participant. The accuracy and repeatability of the measurement are affected by many factors, including the size of the finger and its position over the camera, the stability of the finger during the video recording process, and the lighting level [20]. The latter depends on the ambient light as well as the battery status, which may affect the intensity of the light emitted from the LED during recording. If the level of finger pressure on the camera is high, the blood flow through the finger artery is restricted, resulting in pulse rate miscalculation. To assess the efficacy of our application, we extracted two types of PPG signals and compared them, named frame averaging signal and RSM signal. The frame averaging method is the most common method for extracting PPG signals from CIS sensors. It works by spatially averaging the intensity level A at the pixels of the entire frame or a pre-selected region of interest within the recorded image frame. In addition, we examined the morphological attributes of the acquired signal relative to the PPG benchmark. Within this framework, we used two different analysis methodologies: ensemble average-centered morphological assessment and beat-by-beat morphological assessment. The analysis used was to determine the Signal to Noise Ratio (SNR). B. Application Architecture As mentioned in Figure 1, the first stage is to build the signal measurement application. Many algorithms have been explored for estimating heart rate (HR) and blood oxygenation (SpO2) [11]. provides a comprehensive description of the specific algorithm they used, which involves detecting local peaks of the pulsating signal and utilizing a simple moving average filter with seven coefficients to remove noise. Their findings showed that 87.5% of cases resulted in relative errors of less than 5%, while the other 12.5% of cases showed relative errors exceeding 5%. Similarly, [12] conducted a similar study and utilized sliding average and Fast Fourier Transform (FFT) techniques to extract the heart pulse. Their proposed approach achieved a remarkable AMELIA ROSE SMITH 5 Amelia Rose Smith accuracy of 98% with an error margin of ±2%. In contrast to the approach of Ding et al. [5], which involves the use of Butterworth and Savitzky-Golay filters to decompose each color channel and the subsequent application of a convolutional neural network (CNN) to improve the SpO2 prediction algorithm, we opted for a signal processing-based methodology to estimate heart rate (HR) and blood oxygenation (SPO2). This decision was mainly driven by the fact that the application of CNN algorithms requires a very long training time, thus making it impractical, especially for smartphones equipped with lower-performance processors. Therefore, we focus on signal processing-based techniques, which offer a more feasible and efficient approach to achieve the goal. A number of studies have explored the important role of sampling rate values in influencing estimation results. They have highlighted that the use of a low sampling rate (Fs) for Photoplethysmogram (PPG) can result in suboptimal calculations. For example, [15] conducted a study using a noncontact PPG recorded at 200 frames per second (fps) and underlined the potential influence of low Fs PPG on Pulse Rate Variability (PRV) analysis. They argued that the application of an interpolation approach is necessary to improve the precision of pulse-to-pulse interval timing estimation. In addition, a recent investigation by [21] uncovered major differences in time-domain and frequency-domain PRV analysis metrics under low Fs PPG conditions. Moreover, in the work of [22], it was shown only features extracted from pulse wave measurements with PPG Fs exceeding 30 Hz have substantive relevance for physiological monitoring. The majority of Continuous Photoplethysmogram (CPPG) data is acquired at low Fs, thus causing limitations in feature extraction due to the disturbed morphology of the CPPG signal. Therefore, the scope of cPPG analysis is still limited to basic Heart Rate (HR), Interval Interval (IBI) analysis (time domain and frequency domain Heart Rate variability), and blood oxygen saturation (SpO2) assessment. AMELIA ROSE SMITH 6 Article Study Case Capture the frame with the shutter Extracts color components in each frame Peak Detection Figure 1. Video Processing Method We outline the design of a smartphone application based on signal processing, utilizing the video captured by the rear camera of the smartphone. We utilize the rear camera to record a video of the subject at the fingertip, and then process the RGB component signals of the video data using various methods to estimate heart rate and blood oxygenation. Figure 1 provides an overview of the video processing steps involved in our proposed technique. Initially, we use rolling shutter technique and SavitzkyGolay filter to amplify the signal and reduce noise. After obtaining the pre-processed signal, we apply local maximum and gradient approximation to the signal, thus allowing us to identify individual peaks, which is important for calculating the pulse rate. In addition, we decomposed the investigated signal into DC (direct current) and AC (alternating current) components using the local maximum method, thus allowing us to calculate the ratio of AC and DC components for the red signal. Next, we used Fast Fourier Transform (FFT) to extract the dominant frequency component of the signal, which corresponds to the heart rate, from the sifted red signal. In addition, we used the FFT strategy to calculate the ratios of the signals, followed by linear or quadratic regression analysis to obtain SpO2. AMELIA ROSE SMITH 7 Amelia Rose Smith The initial stage of the system involves applying the rolling shutter technique, which consists of a series of operations. The recorded video is split into individual image frames, with each frame undergoing sequential scanning operations across its pixel rows to collect temporal data attributes such as trow, tf 2f, and corresponding timerelated details for each pixel row. After that, the individual color channels (red, green, and blue) of each frame are separated. In line with recent investigations [1-3], only the red channel (r-channel) is used for this analysis. Next, the luminance values at pixel row i and pixel column y for each image frame, represented as 𝐴(𝑖,𝑗,𝑘), are computed. Each column j in each row i is averaged, taking into account the specified output resolution (we used 360p × 240p), and this process generates k values according to the number of channels selected, resulting in 𝐴(240,1,1). Next comes the stage of increasing the signal sampling rate by starting from the Signal to Noise Ratio (SNR) metric. A subset of pixel rows, denoted as θ, is selected from the total pixel rows, guided by the (SNR) metric as follows: We calculate the ratio between the energy covering the fundamental frequency along with the first harmonic of the pulse signal (SN1) and the residual energy present in the spectrum (SN). In this study, θ is assigned a value of 5, which leads to the selection of five outputs from the 240 input signals obtained in the previous step. After this selection, the pixel rows were reconstructed into a 2nd-order exponential model with an RMSE of 1.33 and R2 of 0.99. Then the signal is amplified using a special amplification algorithm as follows: Where Y(j) is the amplitude of the high-speed signal processed using the amplification algorithm: a, b, c, and d are amplification coefficients, z is the normalization of pixel rows after reconstruction, 𝜓 is a weighting parameter with 0 < 𝜓 < 1 calculated by. AMELIA ROSE SMITH 8 Article Study Case Where ∆𝑇 is the sampling time interval and 𝜏 is a constant. From the algorithm, the signal has a high sampling rate resembling conventional tools. To overcome the noise generated by various background elements such as high-frequency modes, extraneous light sources, and substandard camera quality, a Savitzky-Golay filter is used to smooth the recorded data. The Savitzky-Golay filter uses a moving endela of size 20 to calculate a polynomial fit through least squares regression over neighboring data points obtained from the moving average filter. Previous research by [11] has confirmed the high efficacy of the 20-element sliding window SavitzkyGolay filter in managing rapidly fluctuating data sets. Fast and accurate heart rate calculation, a well-defined oscillatory signal is generated. Two equivalent methods are applied to determine the pulse rate, both focusing on identifying signal peaks and estimating their frequency within each window. The Local Maximum method identifies the peak as a local maximum, while the Gradient method assesses the gradient at neighboring locations around the destination point. A peak is confirmed if the gradient is positive at three consecutive points before the peak and negative at three consecutive points after the peak. To correlate the frequency component of the signal with the pulse rate, Fast Fourier Transformation (FFT) is applied to the smoothed average intensity signal. This analysis revealed dominant peaks in the frequency spectrum, which aligned with the heartbeat frequency. The pulse rate is calculated by multiplying the peak frequency (fp) in Hz by 60. At the next stage, the signal is separated into AC (alternating current) and DC (direct current) components using the Local Maximum method. This involves dividing the signal into frame windows with a distance of two consecutive local minima between each window. The average signal value between the two local minima determines the DC component of each frame window, while the local maximum occurring between the two local minima determines the magnitude of the AC component. The numerical code applies a sliding frame window to calculate all AC and DC components for the red and green spectral components. The blue component, due to its low correlation with heart rate as shown in the Results section, was not considered in this study. The ratio (R) was calculated from the AC and DC components, determined at two different wavelengths (𝜆1 and 𝜆2) around the isosbestic point. The red and green (broadband) components of the video signal were used for the calculation of R, as AMELIA ROSE SMITH 9 Amelia Rose Smith presented in Equation 1. The concentration of oxygen in the blood, SpO2, can be derived from the R-ratios of the AC and DC components. Linear or quadratic regression analysis is commonly used to determine SpO2, with the R ratios serving as independent variables directly related to blood oxygenation. In the next stage, the application is made using Android Studio. The process begins with the initial appearance of the application then proceeds to make the selection of wanting to know the condition of Heart Rate, Oxygen Saturation Level, or All Vital Sign. After that it will continue by providing an explanation of the placement of the finger on the camera. After that, video processing is carried out like the algorithm described earlier. Video processing is done for 30 seconds. After completion, the results will be displayed according to what was previously selected, namely Heart Rate, Oxygen Saturation Level, or All Vital Sign. All Vital Sign is a display of the results of Heart Rate and Oxygen Saturation Level into one. AMELIA ROSE SMITH 10 Article Study Case Start Options: Heart rate, Oxygen saturation level, all vital signs Explanation of finger placement on the camera Video processing for 30 seconds Result display Finish Figure 2. Flowchart of Smartphone Application . RESULTS AND DISCUSSION This study proposes a targeted RSM method to enable signals with high sampling rates on smart phone cameras with low FPS. Before starting the experiment, we conducted some validation studies related to the utilization of color channels in camera sensors. As mentioned earlier, three output channels in different optical wavelengths are obtained in the camera sensor: red, green, and blue channels. On the other hand, the reference PPG signal utilizes the infrared optical wavelength to record the PPG signal. Therefore, to validate the RSM signal with the reference PPG signal, we compared each color channel with the reference PPG. We found that the red channel of the camera sensor showed the most similar morphology compared to the reference device with a correlation coefficient r of 0.99. Therefore, we reasoned that it was natural to compare the red channel of the camera sensor with the reference PPG since both produce similar PPG signal characteristics. In addition, we conducted AMELIA ROSE SMITH 11 Amelia Rose Smith a preclinical study on 30 recruited subjects with an age range of 20-30 years. We measured Pulse Rate (PR) and blood oxygen saturation (SpO2) simultaneously using a medically approved oximeter while recording video footage of the index finger placed directly over the smartphone camera sensor. The next section studies the comparative analysis of the recorded signal morphology compared to the benchmark PPG signal. The main focus is to explore the improved efficacy achieved by increasing the sampling rate through the utilization of the rolling shutter technique. To demonstrate the necessity of Rolling Shutter Modulation (RSM) in achieving high-quality bio-signal acquisition, we compare the RSM signal with the frame-averaged CPPG signal. As mentioned, frame averaging is the most commonly used method for extracting CPPG signals from CIS sensors. Figure 3 presents the signal comparison between the RSM signal and the frame average regarding the reference PPG signal. To ensure a fair comparison, we resampled the frame-averaged CPPG signals from 30 to 150 Hz using linear interpolation. In our initial analysis, we examined the signal comparison from the perspective of waveform morphology. Figures 3(a) and 3(b) show obvious differences in the detection of systolic and diastolic CPPG peak points. The peak point of the RSM signal is aligned with the peak point of the reference signal, while the peak point of the frame average signal has shifted. In Figure 2(b), it is clear that the detected peak of the RSM signal is in the same time range as the reference PPG signal. The significant difference in the peak points of the two signals affects the information value extracted from the signals. Next, we calculate the Signal-to-Noise Ratio (SNR) to compare the signal quality between the traditional method and the rolling shutter method. The RSM signal shows an SNR of 20.05 dB, surpassing the frame average signal with an SNR of 15.7 dB. The resulting RSM signal has a better waveform because in the RSM method each row of pixels goes through the SNR calculation stage, which calculates the ratio of the energy around the fundamental frequency plus the first harmonic of the pulse signal and the remaining energy contained in the signal spectrum. As mentioned earlier, the pixel row with the best ratio calculation is selected and amplified with the amplitude compensation model. AMELIA ROSE SMITH 12 Article Study Case Figure 3A. Signal comparison of frame averaging method and RSM method with PPG reference signal Figure 3B. Signal comparison of frame averaging method and RSM method with PPG reference signal for 1 waveform We can see that the proposed method overcomes the common problem of CPPG which is only useful for simple HR analysis. The proposed method is able to increase the sampling of CPPG data with Fs equivalent to five times the camera fps (Fs = 150 Hz). We have discussed the limitations of the proposed Rolling Shutter Modulation (RSM) method and outlined potential directions for future research. In this study, we focused on testing time-based morphological features to assess the overall morphology of the signal compared to the AMELIA ROSE SMITH 13 Amelia Rose Smith reference PPG signal. To provide a more comprehensive evaluation of physiological measurements, it is important to analyze other features in the frequency and energy domains. Although this manuscript concentrates on enabling smartphones to generate PPG signals that are highly correlated with the RSM method, additional experiments are needed to validate RSM signals for a wide array of assessments, but not limited to arterial stiffness, stress monitoring, and uncuffed blood pressure monitoring. In addition, the data seen in table 1 shows that 30 subjects were measured by comparing the results of the application with the measurement results of the existing SPO2 tool. The results of Heart Rate and Oxygen Saturation Level show that the Average Error of the application in measuring Heart Rate is 1.97 and measuring Oxygen Saturaion Level is 0.87. The average error result of using an OPPO A83 smartphone is 2.18 for Heart Rate and 0.84 for Oxygen Saturation Level. The Average Error result of using Samsung A12 smartphone is 1.12 for Heart Rate and 0.41 for Oxygen Saturation Level. Figure 4A. SNR analysis on frame averaging method signal AMELIA ROSE SMITH 14 Article Study Case Figure 4B. SNR analysis on the RSM method signal The application that has been made is shown in Figure 4(a) which shows the Menu view when choosing to find out the Heart Rate, SP02 or All Vital Sign. Then Figure 4(b) shows the display of finger alignment instructions on the camera, after the finger is correctly positioned then video processing is carried out as in Figure 5(b) using the proposed method. After 30 seconds the results will appear as in Figure 6. In the next stage of research, it is important to conduct further experiments to assess the impact of external interference on the acquired signal quality and the subsequent impact on feature extraction. In addition, one of the methods for PPG-derived physiological monitoring on mobile devices relates to blood pressure monitoring systems [21], [22]. Therefore, for future research, we aim to investigate the potential of the proposed RSM signal to estimate blood pressure in a continuous and cuffless manner using only the built-in CIS on smart phones. Table 1. Measurement Results of Application with Device AMELIA ROSE SMITH 15 Amelia Rose Smith AMELIA ROSE SMITH 16 Article Study Case For What? Attention Figure 4. (a) Selection Menu Display, (b) Instruction Display for placing finger on camera AMELIA ROSE SMITH 17 Amelia Rose Smith Figure 5. (a) Display when the finger has not been attached to the camera, and (b) Display when the finger has been attached to the camera and video processing has been done. Result Result Figure 6. (a) Heart Rate result display, and (b) Oxygen Saturation Level Result Display. AMELIA ROSE SMITH 18 Article Study Case CONCLUSION Several methods were performed to estimate the best HR and SpO2 continuously. The first stage was the preprocessing stage, namely the Rolling Shutter Modulation method and the Savitzky-Golay filter to increase the sampling rate of the acquired CPPG signal and mitigate noise. Based on the experimental results, the proposed method (RSM) can perform well in improving the sampling of CPPG data with Fs equivalent to five times the camera FPS (Fs = 150 Hz). The best SNR of 20.05 dB was obtained using the RSM method. These results show that RSM is recommended in fast and accurate calculation of HR and SpO2 based on fingertip video recordings using consumer-grade smartphones. The results of this application have an average error of 1.97 when measuring heart rate and 0.87 when measuring oxygen saturation. The average error using the OPPO A83 smartphone is 2.18 per heart rate and 0.84 per oxygen saturation. The Samsung A12 smartphone had an average error score of 1.12 for heart rate and 0.41 for oxygen saturation. Involving more analysis of other features in the frequency and energy domains as well as additional experiments are needed to validate theRSM signal for diverse types of assessments, including but not limited to arterial stiffness, stress monitoring, and uncuffed blood pressure monitoring. AMELIA ROSE SMITH 19 Amelia Rose Smith REFERENCES AMELIA ROSE SMITH 20 Article Study Case AMELIA ROSE SMITH 21