The objective of this study was to review consumer-based websites on menopause education for quality, readability, and accuracy of content. A systematic review was conducted using an eight-step process for reviewing consumer-based websites established in prior literature. This work adds to the growing body of literature focused on menopause education and can be reproduced by nurse practitioners who desire to further increase the knowledge of quality, readable web pages for consumer-based education about menopause.
Nurse practitioners (NPs) who care for women are often asked by patients for information about menopause, which is a normal, natural, and expected stage of a woman’s reproductive life span. Most women are living at least a third of their lives post menopause, and yet they are often unprepared for the menopause transition.1,2 Symptoms of menopause are varied, but women who experience symptoms report an associated negative impact of menopause on quality of life.2 Research shows that women who receive education about menopause and its associated symptoms report more positive attitudes about the transition along with less severe symptoms.1,3,4 Women report receiving most of their information about menopause from online sources, friends and family, or from healthcare providers, demonstrating that access to current and accurate information is not readily accessible to them outside of a provider visit.2,3 Most women report not having access to accurate menopause information, and many times the only information they can find online is about hormone replacement.5,6 Given that women report better attitudes about menopause when they have had prior education, the question arises: Are online educational resources an effective means to provide women with information about menopause? A second question follows: Is a systematic approach to evaluating websites an effective way for NPs to differentiate which resources are beneficial? Following the process set forth by researchers in 2018, this website review analyzed the quality of educational websites that provide information about menopause to consumers.7 Knowing which websites are accurate and accessible will give NPs other resources to provide patients who are seeking information about the menopause transition. This is not an exhaustive review but seeks to mimic search method habits as defined by researchers and the evaluation of not only webpages but also the review process.
According to a 2018 Gallup survey conducted by the North American Menopause Society (NAMS), 64% of women get their information about menopause from a source other than their healthcare provider. Additionally, women who have not yet gone through the transition were even less likely to get their information about menopause from their healthcare provider.8 NAMS recommends discussions with all middle-age women about menopause symptoms and treatments, physiologic changes, routine screening recommendations, disease risk-reduction strategies, and midlife psychosocial issues. Starting these discussions during middle age before the menopause transition improves decision making among midlife and older women.9 Even with the NAMS recommendation, menopause anticipatory guidance is not being routinely conducted, leaving women to gather their information about menopause from other sources.8,10
Although access to education about menopause is important, so is the quality of that education. When NPs offer knowledgeable, empathetic, kind, and considerate care while giving information to women about menopause, women are more receptive and report better attitudes toward the information.2,5,9,10 Unfortunately, questions arise outside of women’s healthcare visits, so individuals are left to seek out answers on their own. If a person is seeking information on the internet about menopause, are they finding accurate information that is at a reading level they can understand and does it address all their concerns?
The internet is used by the great majority of adults, with an estimated 70% using this resource for health information.6,11 Interestingly, women are more likely than men to seek health information on the internet.12 Understanding the quality of available online menopause informational sites will help NPs guide women to accurate sources of information to augment that which they receive during the visit.7 In addition to understanding the quality of the information, readability is an equally important factor. Almost half of the United States adult population reads below an 8th-grade reading level. Thus, ensuring information is presented at a level that the average adult can read and comprehend is as important as the quality of the information.13 Most women’s healthcare organization websites, including those from the American College of Obstetricians and Gynecologists, have readability scores geared toward advanced readers at a level above the 8th grade.13 Recognizing consumer-based websites that meet these standards will give NPs accurate and accessible online resources to recommend to women who seek more information about menopause outside of the healthcare visit.
Study aim
The aim of this study was twofold. The first aim was to review the extant online resources of consumer-oriented menopause websites to assess their accuracy, readability, and completeness. The second aim was to simultaneously evaluate a process for clinicians to use for their own assessment of consumer-based websites.
Methods
A systematic review was conducted using a process for reviewing consumer-based websites established in prior literature.7 This process was developed to address identified inconsistencies and incompleteness in previously established evaluation methods. It provides a blueprint for nurses to use when evaluating web-based resources used in conjunction with patient education.7 Using a blend of established processes and Preferred Reporting Items for Systematic Reviews and MetaAnalyses (PRISMA) guidelines, researchers developed a method for conducting a systematic review and website analysis that consists of eight steps: select topic; determine the purpose of analysis; select the terms and search engines to be used in the review; develop and apply inclusion and exclusion criteria for websites; develop processes and tools to manage search results; specify measures of quality to be evaluated; compute readability; and conduct analyses of websites.7 This review follows their framework. The completion of the first two steps is described previously in this article. This section discusses the next four steps that constitute the methods of the study.7
Select search terms and engines
Two search engines were used: Google and Bing.7,14 Google and Bing were chosen specifically because, of the top five search engines, Google accounts for the vast majority of market share usage at 88.06%, with Bing having the second largest market share usage at 6.94%. The remaining three search engines combined constitute less than 5.00% of market share usage.15 Selected search terms were menopause, menopause symptoms, and perimenopause. Due to the risk of search engine optimization and cookies altering the search, computers at the Rochester Hills Public Library were used to mitigate these risks.11 The library wipes cookies clean after each log in and out of a desktop computer. The computer was logged out of and back into to start each search without cookies interfering.
Develop and apply inclusion and exclusion criteria for websites
Most consumers now use their mobile devices to access the internet, and most do not scroll through results. The first several links provided by the search engine are more commonly the ones chosen.16 To conduct the search in a manner that a consumer would, the first three eligible web pages listed on each search were included.
The inclusion criteria were that the website was a consumer-oriented website, menopause was the topic, and the information was written in English. Exclusion criteria included websites that were scientific articles, videos, advertisements, or news reports. The process and outcomes of website selection are shown in the Figure.
Develop processes and tools to manage search results
A screenshot of each search result listing was created and saved as a pdf file. The date and time of each search was recorded. A table was then created to compile the results of the website review including reading level, readability, and the DISCERN measures, as further explained in the following section.
Specify measures of quality to be evaluated
Websites were evaluated for Health on the Net (HON) Foundation certification. Until it was dismantled in December of 2022, HON certification provided assurance that the website creators abide by ethical standards and was a trustworthy indicator of the credibility of the information.7,17,18 The quality of the information presented by the websites was evaluated using the DISCERN tool. Developed in 1999, this tool was designed to be used by consumers and healthcare providers to evaluate the quality of written healthcare information. It has been validated with good interrater reliability.19 DISCERN has been used in several studies to assess the quality of healthcare information on the internet.14,20,21 The tool includes 16 questions that may be rated on a scale of 1 to 5 addressing reliability, quality of treatment descriptions, and an overall rating of quality. Instructions are provided with each item to aid the user in understanding and implementing the tool. Questions 1 to 8 are intended to determine the reliability of the source and its trustworthiness. Questions 9 to 15 address the details of treatment choices provided. The last question uses a combination of the previous questions’ scores and the reviewer’s best judgment to rate the source’s quality.19 For this study, an average score of the preceding questions was used to find the final rating.
To assess for readability, the Flesch-Kincaid grade level and the Flesch Reading Ease (FRE) (as recommended by National Institutes of Health) was implemented utilizing the readability tool on Microsoft Word.7,13 The text from the webpage was copied and pasted into a Word document and the readability tool applied from there. The Flesch-Kincaide grade level scores text and assigns a corresponding grade level based on syllable count and sentence length. The FRE score uses the same criteria as the Flesch-Kincaide scores and additionally includes the number of sentences in the piece of writing.13
Findings
The aim of this study was to determine the top web pages based on four criteria in accordance with the 2018 researchers’ method.7 The complete findings are outlined in the Table. First considered were those web pages that are HON certified. Second, the readability was assessed as indicated by the FRE readability of at least 60. Third evaluated was the reading grade level, with a requirement of a reading grade level no higher than 8th based on the Flesch-Kincaid. The fourth and final aspect evaluated was quality of information as determined by the DISCERN rating, with a rating of 4 or 5 indicating high quality. Three websites held an HON certificate: Healthline.com, WebMD.com, and MayoClinic.com. Of these three, only one met the baselines established by this study: WebMD, with an FRE of 63.8, an FK-GL of 6.9, and a DISCERN of 4.75. Although not HON certified, Womenhealth.gov scored second highest with an FRE of 64.4, an FK-GL of 7.3, and a DISCERN of 4.
Discussion of findings
The aims of this study were met by identifying consumer-based websites about menopause that are accurate and have an ease of readability to make them useful and comprehensible by the general public. All of the websites except Women-shealth.gov and WebMD.com had FK-GL scores over 8, making them inaccessible to the almost half of US adults who read at or below the 8th-grade level. These same websites also had DISCERN scores below 4, discrediting the information contained therein. As a result, the websites Womenshealth.gov and WebMD.com were selected as the two superior webpages for consumer information about menopause. Both Womenshealth.gov and WebMD.com are reliable resources that NPs can provide for their patients who are seeking information about menopause outside of an office visit.
Limitations and future directions
This study was potentially limited by one reviewer of the DISCERN scale. In future iterations, a second or third reviewer could add to the credibility of the rating in some measures while accounting for interrater reliability. Furthermore, using this scalable framework, future replications could increase the number of websites reviewed. HON certification is no longer available. Other similar benchmark criteria such as that used by the Journal of the American Medical Association could be substituted.7 Finally, the websites’ algorithms, which could include cookies or search engine optimization, can alter search results.
Implications
The findings of this study are in agreement with and highlight previous studies that have shown there is a lack of quality information about menopause for the general public. The fact that only two consumer-based webpages met the quality and readability measures assessed in this study underscores the need for more evidence-based menopause education on the internet. Filling the menopause information chasm presents opportunities for researchers and providers to add to the body of evidence and improve access to evidence-based educational resources. This study gives NPs and other healthcare providers two resources that they can recommend to their patients and families who are looking for more menopause information outside of the provider visit: WebMd.com and Womenshealth.gov. This study used a limited sample, but the process outlined in this article could be used by NPs to evaluate any web page.
Anticipatory guidance about menopause should be provided to all women who are approaching midlife.9 By doing so in a collaborative manner, NPs can help women feel more comfortable and empowered in their healthcare choices. Offering other sources of information that can be accessed outside of the office visit can help increase the self-efficacy and empowerment of midlife women as they navigate their menopause years.9,14,20,21 NPs can use the DISCERN scale to easily and quickly assess the quality of written information in web pages, smartphone apps, and even books so that they can guide patients to the best sources of information for many healthcare concerns while weeding out those sources that are less reliable, readable, or accurate. Readability can be easily assessed by copying and pasting the written text into a Microsoft Word document and utilizing the readability statistics tool. Using the method implemented in this study, future researchers can add to the body of knowledge about which online sources of healthcare information are worthy of recommendation or are not.
Conclusion
The findings of this study support the lack of menopause information available to women outside of friends, family, and visits with their healthcare provider.3 With 70% of adults getting healthcare information on the internet, knowledge of quality websites is crucial when referring patients to other sources for information.6 There is a need for innovative, quality, evidence-based online educational resources for menopause information available to all interested persons.
Roxanne B. W. Ropponen is a PhD candidate at Oakland University in Rochester, Michigan. The author has no actual or potential conflicts of interest in relation to the contents of this article.
Womens Healthcare. 2024;12(1):35-40. doi: 10.51256/WHC022435
References
Ensan A, Babazadeh R, Aghamohammadian H, Aghaei MF. Effect of training based on choice theory on health-promoting lifestyle behaviors among menopausal women. J Midwifery Rep Health. 2018;6(2):1253-1263. doi: 10.22038/JMRH.2018.10469
Huang DR, Goodship A, Webber I, et al. Experience and severity of menopause symptoms and effects on health-seeking behaviours: a cross-
sectional online survey of community dwelling adults in the United Kingdom. BMC Womens Health. 2023;23(1):373.
de Arruda Amaral ICG, Baccaro LF, Lui-Filho JF, et al. Opinions and main sources of information about menopause among middle-aged Brazilian women. Menopause. 2019;26(10):1154-1159.
Khandehroo M, Sany SBT, Oakley D, Peyman N. Health literacy intervention and quality of life in menopausal women: a randomized controlled trial. Int J Health Promot Educ. 2020;6(2):114-126. doi: 10.1080/14635240.2020.1762502
Warke B. Designing interactive technological interventions for menopausalwomen: designing and developing interactive technology tools to help aging women navigate information about stages of menopause to increase self-awareness of biopsychosocial changes and manage lifestyle for an improved quality of life. Proceedings of the 15th International Conference on Tangible, Em-bedded, and Embodied Interaction, February 2021. doi: 10.1145/3430524.3443692
Yeganeh L, Boyle JA, Johnston-Ataata K, et al. Positive impact of a co-designed digital resource for women with early menopause. Menopause. 2022;29(6):671-679.
Rew L, Saenz A, Walker LO. A systematic method for reviewing and analyzing health information on consumer-oriented websites. J Adv Nurs. 2018;74(9):2218-2226.
Utian WH, Schiff I. NAMS-Gallup survey on women’s knowledge, information sources, and attitudes to menopause and hormone re-placement therapy. Menopause. 2018;25(11):1172-1179.
Santoro NF, Coons HL, El Khoudary SR, et al. NAMS 2021 Utian Translational Science Symposium, September 2021, Washington, DC. Charting the path to health in midlife and beyond: the biology and practice of wellness. Menopause. 2022;29(5):504-513.
Munn C, Vaughan L, Talaulikar V, et al. Menopause knowledge and education in women under 40: results from an online survey. Womens Health. 2022;18:17455057221139660.
Lewandowski D, Sünkler S, Yagci N. The influence of search engine optimization on Google’s results: a multi-dimensional approach for detecting SEO. Proceedings of the 13th ACM Web Science Conference, June 2021. https://doi.org/10.1145/3447535.3462479
Cohen RA, Adams PF. Use of the internet for health information: United States, 2009. NCHS Data Brief. 2011;66:1-8.
Fahimuddin FZ, Sidhu S, Agrawal A. Reading level of online patient education materials from major obstetrics and gynecology societies. Obstet Gynecol. 2019;133(5):987-993.
Hirsch M, Aggarwal S, Barker C, et al. Googling endometriosis: a systematic review of information available on the internet. Am J Obstet Gynecol. 2017;216(5):451-458.e1.
Search Engine Market Share United States of America. StatCounter Global Stats. https://gs.statcounter.com/search-engine-market-share/all/united-states-of-america.
Ankalkoti P. Survey on search engine optimization tools & techniques. Imperial J Interdisc Res. 2017;3(5):40-43.
HONcode certification status. Drugs.com. Updated November 12, 2023. https://www.drugs.com/support/honcode.html.
Research guides. Health Literacy: Reputable sources. Updated April 20, 2022. https://libraryguides.unh.edu/health-literacy/reputable.
Charnock D. Shepperd S; DISCERN. http://www.discern.org.uk/index.php.
Daraz L, Morrow AS, Ponce OJ, et al. Can patients trust online health information? A meta-narrative systematic review addressing the quality of health information on the internet. J Gen Intern Med. 2019;34(9):1884-1891.
Murtaza F, Shirreff L, Huang LN, et al. Quality and readability of online health information on menopausal hormone therapy in Canada: what are our patients reading? Menopause 2021;29(1):54-62.
KEY WORDS: menopause, menopause education, menopause symptoms, patient education